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Program description 


University of Chicago School Mathematics Project (UCSMP) 
Algebra, designed to increase students’ skiiis in algebra, is 
appropriate for students in grades 7-10, depending on the 
students’ incoming knowiedge. This one-year course highiights 
appiications, uses statistics and geometry to develop the 


algebra of linear equations and inequalities, and includes 
probability concepts in conjunction with algebraic fractions. 

The curriculum emphasizes graphing, while manipulation with 
rational algebraic expressions is delayed until later courses. This 
curriculum uses the UCSMP textbook. 


Research 


One study of UCSMP Algebra first edition, comparing it with 
Saxon Math, and one study of UCSMP Algebra second edition, 
comparing it to traditionai curricuia, met the What Works Clear- 
inghouse (WWC) evidence standards with reservations. 


The two studies included more than 200 middle- and high- 
school students in four rural, suburban, and urban districts in the 
West, Midwest, Northeast, and South.^ 



EffSCtiVGnsSS UCSMP Algebra was found to have potentially positive effects on math achievement. 



Math achievement 

Rating of effectiveness Potentially positive effects 
Improvement index^ Average: +13 percentiie points 

Range: -5 to +31 percentiie points 



1. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 

2. These numbers show the average and range of the improvement index for all findings across the studies. 
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Additional program 
information 



Research 



WWC Intervention Report 



Developer and contact 

Developed by the University of Chicago School Mathematics Proj- 
ect (UCSMP). Address; 6030 S. Eliis Avenue, Chicago, IL 60637. 
Web: http://social-sciences.uchicago.edu/ucsmp/Secondary. 
htmi. Teiephone: (773) 702-1130. UCSMP Algebra 2nd edition is 
distributed by Pearson, Prentice Mali. Web: http://www.phschool. 
com/atschool/ucsmp/index.html . Email: Communications® 
pearsoned.com. Teiephone: (800) 848-9500. The third edition wiil 
be availabie from the Wright Group/McGraw-Hiil. 

Scope of use 

UCSMP Algebra first edition was deveioped and tested between 
1985-88; the second edition was deveioped and tested between 
1992-94.® The third edition has been in deveiopment since 
2004. According to the developers at the University of Chicago, 
since 1983, 3.5-4 miiiion students in elementary, middle, and 
high schoois have been using UCSMP materiais and various 
curricula. Scope of use and demographic characteristics of the 
students using the UCSMP Algebra curriculum are not available. 



Teaching 

A typical lesson consists of commentary and explanations 
about the math concepts or skills, followed by applications 
of the concepts or skills taught in a variety of problem sets. 
The textbook includes summary and vocabulary sections to 
promote student awareness of the skills, properties, uses, 
and representations of the math ideas introduced in the 
chapter. The summary section provides an overview of the 
key concepts and skills and lists the relevant terms explored 
in the chapter. Refresher sections offer a set of problems that 
may be used by the teacher for students who are weak in a 
particular area. The textbook also contains progress self-tests 
for students. 

Cost 

A student text book costs $56.97. A teacher resource book titled 
Implementing UCSMP: A User’s Handbook costs $12.97. An 
additional Study Skills Handbook costs $30.97. 



Three studies reviewed by the WWC investigated the effects of 
UCSMP Algebra first and second editions. One study (Peters, 
1992) was a randomized controlled trial with randomization 
problems that met WWC evidence standards with reservations.'* 
A second study (Thompson, Senk, Witonsky, Usiskin, & Kaeley, 
2006) used a quasi-experimental design that met WWC evidence 
standards with reservations. The remaining study did not meet 
WWC evidence screens. 

Peters (1992) included 36 “math talented” students from 
one junior high school in Nebraska. Most of the students were 
Caucasian, and the district was a unique rural and suburban mix 



that borders two large cities. Students in the intervention group 
used the UCSMP Algebra first edition textbook, while students 
in the comparison group used the Saxon Middle School Math 
curriculum. 

Thompson et al. (2006) included 189 students from three 
high schools in the West, Northeast, and South.® The sample 
consisted of a diverse student population, including students 
from ethnic minorities and low socioeconomic status. The inter- 
vention group used the UCSMP Algebra 2nd edition textbook, 
while students in the comparison group used other non-UCSMP 
textbooks that were used by the schools prior to the study. 



3. The second edition includes new emphases on student writing and projects and increased use of technology, particularly using graphing calculators. 
Because the difference between the two editions Is not substantive, the WWC reviewed research on both editions In this Intervention report. 

4. Peters (1992) compared UCSMP Algebra with Saxon Middle School Math. The author Indicates that a random selection of numbers was used to divide 
the 36 participants between the Intervention and comparison groups. But due to scheduling problems with other course offerings, the number of 
students In each group was changed to 17 In the UCSMP Algebra group and 19 In the Saxon Middle School Math group. Therefore, the study meets 
standards with reservations, according to WWC criteria. 

5. Most of the students were from ninth grade, but about 15% of the sample was enrolled In grades 10-12. 
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Effectiveness 



The WWC found UCSMP 
Algebra to have potentially 
positive effects on 
math achievement 



WWC Intervention Report 



Findings 

The WWC review of middle school math curricula addresses 
student outcomes in the math achievement domain. 

Peters (1992) reported no statistically significant differences 
between the UCSMP Algebra group and the Saxon Middle 
School Math group.® The study also examined students’ 
understanding of algebraic components as measured by scores 
on four unit tests and reported that UCSMP Algebra had an 
overall higher mean score than the comparison group (no level of 
statistical significance was reported). Further, the effect size was 
neither statistically significant nor large enough to be considered 
substantively important by WWC criteria (at least 0.25).^ 

Thompson et al. (2006) examined students’ math achievement 
using three outcomes— the High School Subject Tests: Algebra, 
a test of algebra readiness, and a problem-solving and under- 
standing test®— and reported a statistically significant difference 
favoring UCSMP Algebra students over comparison students 
on algebra readiness and problem-solving and understanding. 



The level of statistical significance of those differences was 
confirmed by the WWC. Further, the average effect size across 
all three outcomes in this study was statistically significant, 
according to the WWC analysis. 

In sum, one study of UCSMP Algebra 1st edition showed an 
indeterminate effect, and one study of UCSMP Algebra 2nd edi- 
tion showed a statistically significant positive effect. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings,® the size of the 
difference between participants in the intervention and compari- 
son conditions, and the consistency in findings across studies 
(see the WWC Intervention Rating Scheme) . 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 
an average improvement index across studies (see Technical 
Details of WWC Conducted Computations^ The improvement 
index represents the difference between the percentile rank 
of the average student in the intervention condition versus 
the percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is entirely based on the size of the effect, regardless of 



the statistical significance of the effect, the study design, or the 
analyses. The improvement index can take on values between 
-50 and +50, with positive numbers denoting results favorable to 
the intervention group. 

The average improvement index for the math achievement 
domain is +13 percentile points across the two studies, with a 
range of -5 to +31 percentile points across findings. 

Summary 

The WWC reviewed three studies on UCSMP Algebra. Of these, 
two studies met WWC evidence standards with reservations and 



6. Peters (1992) reported statistically significant growth between pretest and posttest among both UCSMP Algebra and Saxor: Math students. 

7. The Orieans-Hanna prognosis test is typically administered as a measure of students’ basic Algebra skills and does not cover the full scope of the 
knowledge and skills taught in a UCSMP Algebra course. It was accepted for review because it assesses important mathematics knowledge and skills 
relevant to middle school math. 

8. The algebra readiness and problem-solving and understanding tests were constructed by UCSMP. 

9. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 
classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of UCSMP Algebra, corrections for clustering and 
multiple comparisons were needed. 
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The WWC found UCSMP 
Algebra to have potentially 
positive effects on math 
achievement (continued) 



one study did not meet WWC evidence screens. Based on the 
results of these two studies, the WWC found potentiaiiy positive 
effects on students’ math achievement. The evidence presented 
in this report may change as new research emerges. 



References Met WWC evidence standards with reservations 

Peters, K. G. (1992). Skiii performance comparabiiity of two 
aigebra programs on an eighth-grade population. Dissertation 
Abstracts International, 54(01), 77A. (UMI No. 9314428) 
Thompson, D. R., Senk, S. L., Witonsky, D., Usiskin, Z., & Kaeley, 
G. (2006). An evaluation of the second edition of UCSMP 
Algebra. Chicago: University of Chicago Schooi Mathematics 
Project. 



For more information about specific studies and WWC caicuiations, piease see the WWC UCSMP Algebra 
Technicai Appendices . 



Did not meet WWC evidence screens 

Mathison, S., Hedges, L. V., Stodolsky, S., Fiores, R, & Sarther, 
C. (1 989). Teaching and learning algebra: An evaluation of 
UCSMP ALGEBRA (Evaiuation Rep. No. 88/89-ALG-1). Chi- 
cago: University of Chicago.''° 



10. Complete data are not reported: the WWC could not compute effect sizes. 
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Appendix 



Appendix A1.1 


Study characteristics: Peters, 1992 (randomized controiied triai with randomization prohiems) 


Characteristic 


Description 


Study citation 


Peters, K. G. (1992). Skill performance comparability of two algebra programs on an eighth-grade population. Unpublished doctoral dissertation. University of Nebraska, 
Lincoln, NE. 


Participants 


The study included 36 eighth-grade students. All of the students were “math talented” based on teacher recommendations, prior academic achievement, and personal 
maturity. The students scored at the 87th percentile or above on the California Achievement Test total math battery. 


Setting 


The study took place in one junior high school in a rural suburban district abutting Lincoln, Nebraska. Students were randomly^ assigned to one of two classrooms (one 
intervention classroom and one comparison). The same teacher taught both the intervention and comparison groups. 


Intervention^ 


Participants in the intervention group were taught the UCSMP Aigebra curriculum. No information was provided about the fidelity of implementation. 


Comparison 


Participants in the comparison group were taught using the Saxon Middie School Math curriculum for eighth-grade students (Algebra 1/2). Students in this group participated 
in daily sessions for one academic year. In each session, the teacher introduced a new concept incrementally, and students had opportunities to practice the new concept and 
past concepts during each session. Students were assessed every fifth lesson. The Saxon Math curriculum is designed to cover 120 lessons in one year. 


Primary outcomes 
and measurement 


The primary outcome measures are the Orleans-Hanna Algebra Prognosis Test and the Understanding of Algebraic Components test.^ (See Appendix A2 for more detailed 
descriptions of outcome measures.) 


Teacher training 


The study noted that the teacher who taught both study groups did not have prior experience with the intervention or comparison curricula, but read extensively about both 
teaching formats. The teacher participated in a one-week summer workshop on UCSMP Algebra, and in two additional one-day workshops given by local consultants on the 
curricula used in this study. Further, agreed-upon components of both the intervention and comparison curricula were monitored on a weekly basis by the researcher to help 
maintain the integrity of implementation. 



1. The study indicates that a random seiection of numbers was used to divide the 36 participants between the intervention and comparison groups. But due to scheduiing probiems for other 
course offerings, the number of students in each group was changed to 17 in the UCSMP Algebra group and 19 in the Saxon Math group. The study meets standards with reservations because 
baseiine differences were controiled for in the statistical analysis. 

2. The same teacher taught both the intervention and comparison groups. Because there is no indication in the study to assume that the teacher was biased toward one of the conditions, this 
design was accepted for review. 

3. The Orleans-Hanna prognosis test is typically administered as a measure of students’ basic Algebra skills and does not cover the full scope of the knowledge and skills taught in a UCSMP 
Algebra course. It was accepted for review because it assesses important mathematics knowledge and skills relevant to middle school math. 
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Appendix A1.2 Study characteristics: Thompson, Senk, Witonsky, Usiskin, & Kaeiey, 2006 (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Thompson, D. R., Senk, S. L., Witonsky, D., Usiskin, Z., & Kaeiey, G. (2006). An evaluation of the second edition of UCSMP Algebra. Chicago: University of Chicago Schooi 
Mathematics Project. 


Participants 


The study inciuded 189 students (98 in the intervention group and 91 in the comparison group) in six matched pairs of ciassrooms. In some of the participating schools, 
classrooms were randomly assigned to conditions.^ About 85% of students were in ninth grade; the remaining students were enrolled in upper grade levels. A prerequisite for 
participation was a strong pre-algebra course or UCSMP Transition Mathematics in the preceding year. The sample consisted of a diverse student population, including ethnic 
minority students and students from low socioeconomic status. About half of the participants were female (53.1% in the intervention group, 50.5% in the comparison group). 


Setting 


The study participants attended three high schools in the West, Northeast, and South. ^ Two of the high schools were located in suburban districts, and the third school served 
inner-city and suburban populations. 


Intervention 


The intervention group used UCSMP Algebra 2nd edition, which was tested in this study and revised afterwards. Class time spent on instruction ranged 43-58 minutes, with 
a mean of 51 .3 minutes and a standard deviation of 7.6 minutes. The intervention group teachers were expected to cover 12 chapters; one of the teachers was expected 
to cover only the first sections of those chapters. Based on teachers’ reports, the study reported major differences in students’ opportunity to learn different mathematic 
concepts. For example, UCSMP Algebra students studied application of the concepts in the algebra text, while comparison students in two schools had limited exposure to 
applications. Intervention group students had more access to calculators than comparison group students. Neither intervention nor comparison group students had access to 
computers. 


Comparison 


The comparison group used the mathematics textbooks that were in place in the schools prior to the study — Algebra 1: An Incremental Development (Saxon), Algebra: Struc- 
ture and Method Book 1 (Houghton Mifflin), and Algebra 1 (Prentice Hall). Class time spent on instruction was similar to that reported for the intervention group. Comparison 
group teachers were expected to cover 10-11 of the chapters in the textbook; one teacher was expected to cover 12 of the 13 chapters. 


Primary outcomes 
and measurement 


This study used three measures-High School Subject Tests: Algebra, an algebra readiness test, and a problem-solving and understanding test.^ Achievement on the general 
mathematics test and the algebra readiness test was analyzed in three ways: overall achievement using all test items; using only the test items for which all study participants 
in the same school had the opportunity to learn the content needed to answer the items, so the test was different in each school (referred to as “fair test’’); and using only 
the test items for which all students in all participating schools had the opportunity to learn the content needed to answer the items, so the test was the same in all schools 
(referred to as “conservative test”). For the purposes of this WWC review, only the analyses focusing on overall achievement were used for rating purposes. (See Appendix A2 
for more detailed descriptions of outcome measures.) 


Teacher training 


The teachers in the intervention group did receive training before or during the school year. These teachers received the textbook chapters, lesson notes, and answers to 
questions in stages throughout the school year. According to the study, none of the teachers had previously taught using UCSMP Algebra textbooks. No information was 
provided for training of teachers in the comparison group. 



1. The study also reported on a comparison between UCSMP Algebra first edition and second edition and found no statistically significant differences. Because this intervention report reviews the 
evidence for both versions of the curriculum, this comparison was not reviewed for rating purposes. The study did not include a comparison of first edition with a non-UCSMP Algebra compari- 
son group. 

2. Although the study participants attended high schools, this sample is relevant to the scope of the Middle School Math review, which focuses on students in grades 6-9 regardless of the setting 
(for example, middle school, junior high school, or high school). 

3. The algebra readiness test and the problem-solving and understanding test were developed by UCSMP. 
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Appendix A2 Outcome measures in the math achievement domain 



Outcome measure 


Description 


Orleans-Hanna Algebra 
Prognosis Test 


This nationally normed test consists of 60 multiple-choice items based on nine model lessons and tive questionnaire items that require students to report their course grades 
and predict their final grade if they were to take algebra. In contrast to an achievement test, students are required to answer questions by following a prooedure or set of 
operations using mathematical or verbal expressions parallel to but ditterent from those contained in the model lessons. This test is often used to prediot the ability to sucoeed 
in a first-year algebra oourse (as cited in Peters, 1992). 


Understanding of 
Algebraic Components 


Four unit tests designed to examine understanding of 12 algebraic components. The four units focus on algebraic terms and expressions, linear equations, exponents and 
polynomials, and systems, parabolas, and quadratic equations. Across units, this measure ineludes a total of 120 items (as eited in Peters, 1992). A performance average of 
the pereentage of skills mastered by each of the students was used. 


High School Subject 
Tests; Algebra 


This test consists of 40 items (20 arithmetic skills items and 20 arithmetic uses items). The test was developed by Scott, Foresman, and Company (as oited in Thompson 
et al., 2006). 


Algebra Readiness 


This test consists of 11 multiple-ohoioe items and 10 short constructed-response items. The test, construeted by UCSMP, was aocepted for review based on description of its 
content (as cited in Thompson et al., 2006). 


Problem-solving and 
Understanding 


This open-ended problem-solving test was developed by UCSMP (as eited in Thompson et al., 2006). This test was administered in two different forms — ’’even form” and 
“odd form” — and each form examines different mathematics skills and knowledge (for example, probability, mean and median, deeimals and fractions, and area and perim- 
eter). The test was acoepted for review based on description of its content. Flalf of the students in each class were randomly assigned eaeh form type. 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(classrooms/ 
students) 


UCSMP Algebra Comparison 

group group 


Mean difference^ 
{UCSMP 
Algebra - 
comparison) 


Statistical 
significance^ 
Effect size'' (at a = 0.05) 


Improvement 

index^ 







Peters, 1992 (randomized controlled trial with randomization problems)^ 








Orleans-Hanna Algebra 
Prognosis Test 


Grade 8 
(math talented) 


2/36 


95.06 

(4.09) 


95.63 

(4.53) 


-0.57 


-0.13 


ns 


-5 


Understanding of Algebraic 
Components® 


Grade 8 
(math talented) 


2/36 


17.44 

(4,16) 


16.09 

(5.23) 


1.35 


0.28 


ns 


+11 


Average^ for math achievement (Peters, 1992) 










0.08 


ns 


+3 








Thompson et al., 2006 (quasi-experimental design)^ 








High School Subject 
Tests: Algebra 


Grades 9-12 


12/189 


47.90 

(16.30) 


46.00 

(14.90) 


1.90 


0.12 


ns 


+5 


Algebra Readiness 


Grades 9-12 


12/189 


49.50 

(16.30) 


37,30 

(14.90) 


12.20 


0.78 


Statistically 

significant 


+28 


Problem-solving and 
Understanding 


Grades 9-12 


12/189 


6.23 

(3.69)1“ 


3.39 

(2.54) 


2.84 


0.89 


Statistically 

significant 


+31 


Average^ for math achievement (Thompson et al., 2006) 










0.60 


Statistically 

significant 


+22 


Domain average^ for math achievement across all studies 








0.34 


na 


+13 



ns = not statistically significant 

na = not applicable 

1 . This appendix reports findings considered for fhe effecfiveness rafing and the improvement index. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 

between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the 
clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Peters (1992), no corrections for 
clustering or multiple comparisons were needed. In the case of Thompson et al. (2006), corrections for clustering and multiple comparisons were needed, so the significance levels may differ from those reported in the original study. 

8. Means and standard deviations for this student outcome were calculated by the WWC based on raw data presented in the appendices of the original study report. 

9. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect size. 

10. Means and standard deviations for the problem-solving and understanding test were calculated by the WWC based on the means and standard deviations presented for the even and odd forms of this test (see Appendix A4), taking into 

account the number of students who completed each type of form. 
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Appendix A4 Summary of additional findings for the math achievement domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


UCSMP Algebra Comparison 

group group 


Mean difference^ 
{UCSMP 
Algebra - 
comparison) 


Statistical 
significance^ 
Effect size'' (at a = 0.05) 


Improvement 

index^ 









Thompson et al., 2006 (quasi-experimental design)^ 








Problem-solving and 


Grades 9-12 


12/101 


5.20 


2.70 


2.50 


0.88 


Statistically 


-f31 


Understanding — odd form 






(3.30) 


(2.20) 






significant 




Problem-solving and 
Understanding — even form 


Grades 9-12 


12/88 


7.40 

(3.80) 


4.20 

(2.70) 


3.20 


0.96 


Statistically 

significant 


-f33 



ns = not statistically significant 
nr = not reported 



1 . This appendix presents findings for fhe even form and odd form versions of fhe problem-solving and undersfanding tesf reporfed in Thompson ef al. (2006). Aggregafed scores across forms were used for rating purposes and are 
presented in Appendix A3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. Positive differences and effecf sizes favor the intervention group; negative differences and effecf sizes favor fhe comparison group. 

4. For an explanation of fhe effecf size calculafion, see Technical Defails of WWC-Conducted Comoufations . 

5. Stafistical significance is fhe probability that the difference befween groups is a resulf of chance rafher fhan a real difference befween fhe groups. 

6. The improvement index represents the difference between the percentile rank of fhe average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and -t-50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings 
not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducfed Computations for the formulas the WWC used to calculate 
statistical significance. In the case of Thompson et al. (2005), a correction for clustering was needed. 
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Appendix A5 UCSMP Algebra rating for the math achievement domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentially negative, or negative.'' 

For the outcome domain of math achievement, the WWC rated UCSMP Algebra as having potentiaiiy positive effects. It did not meet the criteria for positive effects 
because no studies met WWC evidence standards for a strong design. The other ratings (mixed effects, no discernibie effects, potentially negative effects, and nega- 
tive effects) were not considered because UCSMP Algebra was assigned the highest appiicable rating. 

Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Met. One study showed a statistically significant positive effect. 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. One study showed a statistically significant positive effect, and one study showed an indeterminate effect. No studies showed a statistically 
significant or substantively important negative effect. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No studies met WWC evidence standards for a strong design, and only one study showed statistically significant positive effects. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No studies showed statistically significant or substantively important negative effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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